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SBSTRACT 

Heads of evaluation units in 21 state education 
agencies responded to a structured telephone interview about their 
unit's workload, evaluation methods, and associated problems and 
needs. Titre committed to evaluation ranged from 10& to 100% and did 
not relate to staff size which ranged from 1-72. Eleven states 
contracted some of their evaluations. The dominant evaluation method 
(11 states) involved setting behavioral objectives and using tests 
and an experimental design to assess achievement. Three states used 
the discrepancy evaluation model: four states audited and accredited 
schools: the remaining states selected the method they felt was best 
suited to the program. Commonly cited problems were the lack of 
evaluators, time, funds, training, and particularly the lack of 
impact of evaluations. Difficulties in designing eatper iments , the 
need for school district data management systems, and evaluation 
training of school personnel were also cited. Because the problems 
and constraints of state evaluation units are so diverse, perhaps the 
strategy should be to develop innovative methods and then to 
determine what problems the methods solve. (The telephone survey 
questionnaire is appended^ (CP) 
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PREFACE 



The Research on Evaluation Program is a Northwest Regional Educational 
Laboratory project of research, development, testing, and training 
designed to create new evaluation methodologies for ui*e in education • 
This document is one of a series of papers and reports produced by 
program staff, visiting scholars^ adjunct scholars, and project 
collaborators — all members of a cooperative network of colleagues 
working on the development of new methodologies* 

What methodological problems do state and local evaluation units 
encounter? What types of new methods would be most useful to them? 
The purpose of the field assessment survey was to identify the character- 
istics of innovative methodological developments which would have 
maximum utility for in^roving the evaluations carried out by local and 
state evaluation units • The survey attempted to determine the evaluative 
activities of units as well as the problems, constraints, and conditions 
under which they operate* 
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FIELD ASSESSMENT SURVEY 



Purpose and Rationale 

Most state education departments (SEAs) and large school districts 
(LEAs) have a unit which is wholly or partly devoted to carrying out 
evaluation* These units vary in the nature and scope of the evaluation 
work tliey carry out and the size of their staffs. As well as carrying 
out or supervising evaluations ^ staffs of these units are often also 
involved in activities which may be related or unrelated to evaluation. 

The Research on Evaluation program at The Northwest Regional 
Educational Laboratory has the aim of developing new evaluation method- 
ologies for use in SEA and LEA evaluations. The purpose of this study 
was to ascertain the activities, evaluative methodologies, and problems 
and constraints of LEA/SEA evaluation units • The information obtained 
should be a useful guide in the development of new evaluation methodologies. 

Our ultimate aim was to try to identify the characteristics of 
innovative methodological developments which would have maximum utility 
to improving LEA/SEA evaluations. To have utility, any new method- 
ologies m.\i$it be adaptable to the functions of evaluation activities 
carried out by LEA/SEA units. Therefore information was required on the 
functions of the various evaluation activities carried out by LEA/SEA 
evaluation units. To have utility, ideally new methodologies should also 
help to solve some of the problems that LEA/SEA evaluation units 
^ experience. New evaluation methodologies should be capable of 

utilisation under the constraints imposed by the conditions and settings 
which units work under. Therefore, infoxrmation was required on the 
problems and constraints that units experience. 
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Information on LEA Evaluation Units 



Some information on LEA units was found in the literature. 

Holley (1978) and Stephens and Barber (1978), all members of LEA 

evaluation units ^ writa of the difficulty of trying to serve the various 

clients of an evaxuation, Holley (1978, p. 10) states: "Since the 

evaluation unit really can*t afford to lose touch with any of the 

potential clients, more time will be required to keep all bases of 

communication covered. In many cases, this dictates more staff. . 

Stephens and Barber (1978, pp* 1, 5, 6, 8) state: 

In most school districts teachers have specific things 
that they want from an RD&E unit, while superintendents, 
boards of dir'^ctors, business offices, and personnel 
offices have other sets of expectations. Once an 
RD&E unit defines its clients, that definition limits 
the services available to otlier portions of the 
institution. By limiting the services through 
defining the major clients, cooperation that is 
necessary to carry oui the basic charge may actually 
be inhibited* • . • At this point, we need to 
clarify that our primary service has been to provide 
information to the board and to the superintendent. , . . 
Now at the same time, the largest number of people 
ixx the district, i.e*, classroom teachers, are not 
really being served as our client, and so they, 
through their trade union, have very little interest 
in seeing that the RD&E unit continues. • . . It 
is apparent that, regardless of the primary clients, 
other powerful groups feel neglected if their needs 
are not met. How to meet the variety of needs 
generated from within a school district and remain 
with the limited budget becomes a topic of major 
importance. 

Holley and Lee (1977) describe a number of problems encovmtered 
by a LEA evaluation unit. "Finding someone who appropriately should 
make a decision is one of evaluation's more difficult tasks." Those 
who should take actions on the bas .s of aun evaluation report will not. 
Another problem is that the treatment in a program changes over time 
and this invalidates any experimental design tliat has been set up. 
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One of the biggest changes that occurs in a program is changes in staff 
The authors express their frustration with the lack of impact of evalua 
tion on policy making. One of the problems is how to get decision 
makers to read evaluation reports* 

Polemeni (undated) , Director of the Office of Educational 
Evaluation, Board of Education of the City of New York, mentions a 
number of problems- These are well worth reading and are given in 
Appendix !• 

To summarize, some of the problems of LEA evali^ation units 
mentioned in the literature are as follows. 



(i) The disparity between the extent of evaluation expected 
and the amount of funding provided* 

(ii) The difficulty of trying to serve a variety of clients 
at one time* 

(iii) The lack of cooperation of school personnel in the 
collection of data* 

(iv) Treatment in a program changes with time and this 
invalidates any design that has been set up. 



(i) The unwillingness to publicize evaluation findings 
which have politically undesirable implications. 

(ii) Evaluation results are often not considered in 
maUcing program or management decisions and 
evaluations lack impact on policy making. 

(iii) The problem of communicating evaluation results to laymen. 

(iv) The problem of getting decision makers to read evaluation 
reports, 

A paper by Webster and stuff lebeam (1978) characterizes and 
assesses the different patterns of practice in educational evaluation 



A. Carrying Out An Evaluation 



B* Reporting And Impact 




tkiat have emerged i.n large urban scHlxsI districts during tha past 

decade. Some of the findings of this study are: 

!• The vast majority of evaluation units control the 
testing function ♦ 

2* Input evalxiation is practically nonexistent in medium 
and small evaluation units* 

3. Process evaluation is less emphasized as evaluatir^n 
units become smaller. 

4. Evaluation units, regardless of si2;e, put most efforts 
into testing, product evaluations, and data processing. 

5. The smaller the unit, the greater the amount of 
time that the unit head generally must spend selling 
evaluation activities to decision makers, including, 
in many instances, convincing them that they need 
infojnnation to make better decisions* 

6. Small evaluation departments spend a comparatively 
small amount of resources on ad hoc information 
requirements. 

?• Evaluation methodology and experimental design were 

consistently ranked among the most important competencies 
expected of evaluators in evaluation units. 

8. Objectives -based evaluations were carried on by 
every unit in the san^le, particularly in conjunction 
with process evaluation and utilizing some form of 
experimental design. 

9. Most large urban districts perform an accountability 
fuxxction. 

10* Policy studies are done at one time or another by 
all large urban evaluation departments, 

11. Most district evaluation units concentrate on providing 
data for decision making. 

Two studies concerning LEA evaluation units are at present in 
progress. It is hoped that these two studies will provide the Research 
on Evaluation Program with further information about LEA evalxiation 
units. 
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Frank Chase of the Urban Education Studies in Dallas is conducting 
a national study of the conditions affecting utilization of knowledge 
from research and evaluation in urban school districts* The purposes 
of the proposed study include: 

1. To develop an accurate picture of the amount and 
types of research and evaluation conducted in 
city school systems dur'....g the period 1973-'197a* 

2. To atnalyze the policies and processes governing the 
authorization of, and the quality controls for, 
research and evaluation projects* 

3. To appraise the mechanics and processes for the 
interpretation, communication, and cipplication of 

the knowledge gained to educational decisions and practices* 

The most extensive study of LEA evaluation units is being carried 
out by Lyon (1973) at the Center for the Study of Evaluation at UCLA* 
An extensive questionnaire has been sent to LEA evaluation units through- 
out the nation* Questions being asked include; What are the activities 
of offices of evaluation? How are evaluation unit products being used? 
How are evaluation offices organised? What are the characteristics of 
evaluation personnel? How are offices of evaluation financed? What 
characteristics of school districts affect evaluation offices? What 
resource constraints and requests are reported by heads of evaluation , 
organizations? This study should be a particularly valuable source of 
information on LEA evaluation units* This information should be a 
useful guide to the Research on Evaliiation Program in the development of 
new evaluation methodologies suitable for LEA evaluation units* 

Since extensive work is being done in studying LEA evaluation units, 
it was decided not to duplicate this work, but instead to concentrate 
on SEA evaluation units where nobody appeared to be doing any research 
and little information is available. Consequently this document 



reports on a study of SEA evaluation units throughout the 
nation « 

Information on SEA Evaluation Units 
As indicated^ there is little information available on SEA 
evaluation units* Plog (1978, p. 2), an evaluator in the Illinois 
Office of Education (an SEA) , mentions the constraints of time and money, 
•'State evaluators are often more constrained by time than researchers 
from a university. Constraints of money are related to time constraints. 
A state bureaucracy is concerned about such things as travel costs, 
per diem, and salaries." 

In order to obtain information on SEA evaluation units, twenty- 
five states were contacted by telephone and the heads of evaluation 
were interviewed. These interviews used a set of open-ended questions 
and lasted on the average for twenty minutes • The following pages 
describe the data collection, the results of the interviews and the 
implications of the data. 
The Data Collection 

Data were collected by means of a structured telephone interview 
schedule which is shown in Appendix II. Advice given by Dillman (197S) 
was used in construction of the schedule. The initial versions of 
the schedule were revised on the basis of trials. Initially this was 
done using staff members of the Title I Evaluation Technical 
Assistance Center of the Northwest Regional Educational Laboratory. 
These staff members provide technical assistance on evaluation to LEAs 
and SEAs and hence are familiar with their evaluation units. Staff 
members played the role of respondents, and deficiencies in the schedule 
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were detected and corrected. Next^ the schedule was tried with two 
nearby LEA evaluations unit heads and further revisions were made. 

The interviewing of the administrative heads of SEA evaluation 
units was done in two stages • Initially the thirteen most western 
states of the U.S. were contacted. These were Alaska^ Hawaii^ 
Washington, Oregon r California, Idaho, Nevada, Utah, Arizona, Montana, 
Wyoming, Colorado and New Mexico. The reason for contacting units in 
these states was that these were the units that the Northwest Regional 
Educational Laboratory had most contact with* It was argued that given 
resource limitations, these states would be closest to the Laboratory 
in order to trial innovative evaluation methodologies. Later it was 
decided to obtain an overall picture of the units in the U.S. Thus, 
from the remaining 37 states, a random sau^le of 12 were selected. 
These states were Kansas, Minnesota, Arkansas, Illinois, Michigan, 
Alabama, Florida, Virginia, New York, New Jersey, Massachusetts, and 
Maine. The distribution of the total of 25 states that were chosen 
are shown on the map of the United States as Figxare 1. 

The names, titles, and telephone numbers of the administrators of 
SEA evaluation units were known for the western states. For the rest of 
the sample, the state education department telephone niimber was known 
only and the administrative head of the evaluation unit had to be 
located. 

None of the administrative heads refused to be interviewed. Their 
cooperation and willingness to fully answer questions were quite out- 
standing. Interviews lasted from fifteen minutes to half an hour. 
Results From the Survey of SEA Evaluation Units 

For four of the twenty-five states the interview schedule was not 
appropriate. This was because these states did not have units that 




carried out program evaluation. The unit head of one state indicated 
that his unit only contracted evaluation studies and did not carry out 
program evaluation studies ♦ The unit head of a second state indicated 
that the unit simply supervised the statewide testing programs and 
carried out no program evaluation. The unit head of a third state said 
they csurried out statewide assessment and were developing a basic skills 
improvement plan. He said that they did not visit schools as local 
autonomy was paramount. In the fourth state, staff members of the 
elementary and secondary divisions did visit schools and sit in on 
classrooms. These were in the nature of inspectorial visits • Reports 
were written but they could not write anything about teachers because 
the teachers* union was so strong* 

In some states, all program evaluations were not necessarily 
carried out by the evaluation unit. At times, other units within a 
state department administration carried out evaluations, especially 
mandated evaluations for Federal programs* 

The following sections describe the answers that were given to each 
of the questions of the interview schedule* 

The nature of the yyork load and size of staffs of units . Answers 
to the questions related to the evaluative work load and the siae of 
staff are shown in Table 1* It is clear from the Table that the 
percentage of time that the staffs of the units spend on program 
evaluation varies markedly from state to state, the range being from 
10 to 100 percent. Unit heads often had difficulty in answering this 
question since it is not always easy to decide what to include in 
program evaluation. Exaimples of borderline activities are: arranging 
testing in tne schools, evaluating facilities such as school buildings. 
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Table 1 



The Evaluative Work Load and Size of Staff 
of the Evaluation Units in 25 State 
Departments of Education 







Average Ho*, ot 
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11 
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50 
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60 
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17 
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25 
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14 
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14 
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7 


20 


100 


6 


92 


4 


21 


100 
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2 


22 
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25 


13-14 


4 


23 


100 


72 


4 


72 


24 


100 


174+ 


0 


I'*- 


25 


100 


300^ 


Moat 


a 



* The respondent found it impose ttola to answer* The respondent felt that they were not 
deeply into evaluation* moet of their evaluations beinqr contracted out* 

♦*Oae person (full*ti«e equivalent) . 

♦ This person mainly serves to tninitor the evaluations carried out by the local school 
districts and this explains the large number of evaluations- 

*4^he evaluation unit in this state does not actually carry out evaluatione but monitors 
and provides guidelines for others (especially local districts) to carry out evaluations* 
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accreditation of schools when this mainly involves filling out 
foriQS. 

For those units not fully engaged in program evaluation activity ^ 
a major activity was planning. Another activity was program development • 
For some units ^ prograin evaluation, planning and development were seen 
to be interconnected activities • Overall^ evaluation units were 
involved in a wide variety of activities including such things as test 
development, studies for departmental policies, accreditation r research 
into funding schemes, insejcvice workshops for school personnel and •'any 
duties that the department requires." 

Table 1 shows that the average number of evaluations conducted per 
year varies considerably from state to state and shows little relation 
to the size of the staff. There is a couple of reasons for this. One 
is that the size of a single evaluation is extremely variable. For 
example, a statewide testing program was counted as a single evaluation, 
A second reason is that there is a variation in the extent to which 
staff iicttially carry out evaluations themselves. For example, some 
units actually monitor or supervise evaluations carried out by others • 
As an example in Table 1, state number 16 conducted 174 evaluations 
which involved one staff member who monitored the evaluations carried 
out by local school districts. Thus^ there is a great variation among 
states in the extent to which evaluation imits are actually involved in 
condxicting evaluation studies themselves. 

Ten states did not contract out evaluations. The rest of the 
states varied greatly in the number of contracted evaluations. 
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The size of the staffs ranged from 1 to 72. The director with 
the 72 full-time staff indicated that his was the biggest state evalua- 
tion unit in the United States. 

In summary, SEA evaluation units show a great diversity in the 
nature and size of their evaluative work load, and the size of their 
staff. Only six units engaged full-time in program evaluation. The 
other units engaged in a variety of activities besides program evaluation. 
The average number of evaluations conducted per year varies considerably 
from state to state and shows little relation to the size of the staff. 
Ten units did not contract out evaluations. The size of the staffs 
ranged from 1 to 72. 

Strategies used in doing an evaluation . In answer to the question, 
•'What strategies (or design) do you use in doing an evaluation?" some 
states indicat-ed raoj:& than one strategy. Eleven states indicated that 
they used an approach which involved identifying objectives and determining 
the extent to which they have been achieved. Associated with this 
appj^oach was the use of testing and the use of axi experimental or 
quasi-experimental design such as involving pre- and post-testing and 
a control group. Stake (1976, pp. 21, 28) calls this approach student 
gain by testing. House (1978, p. 4) calls it the behavioral objectives 
approach. "The objectives of a program are spelled out in terms of 
specific student performances that can be reduced to specific student 
behaviors. These behaviors are measured by tests, ..." Cuba (1977) 
calls it the objectives approach or the Tylerian or neo-Tylerian 
approach, after its first proponent, Ralph Tyler. Three states indicated 
that their major strategy was the discrepancy evaluation model of Malcolm 
Provus (1973), which is a Neo-Tylerian approach. Some state units 
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indicated that the use of the objectives approach affec-ced a program 

so that it was better managed and it could be evaluated. 

Four states are quite distinctive in that they use an auditing- 

accrediting approach to evaluation. While there is variation in the 

four states, all audi ting-accrediting approaches involve a team of 

evaluators visiting the schools. One of the four approaches is heavily 

reliant on the self -evaluations done by schools. The approaches of two 

of the fotir states are described below.* 

The criteria for the evaluations are mtnimm standards set by 
the state. These standards refer not Just to ciirriaulum but to a 
variety of aspects of the fmotioning of the schools . Afe use a team 
approach where all 21 evaluators go into a district. First we meet 
with the district administrators and explain the criteria and the 
evaluative process. Then we meet with teachers to do the same thing, 
Ve use a checklist which is an expanded version of the criteria. Ve 
spend one to three weeks in the schools. For schools that are selected 
in a district^ we visit every classroom in these schools. After evaluation 
is complete we discuss it with teachers and administrators, Approxi" 
mately 20 days after we send back a irritten report and allow SO days 
for any reply. If a district is not incompliance with the standards^ 
they must develop a plan to comply.. This plan is given to the 
accrediting agency and is used by them when they go into the district 
the following year. We work on a six year cycle. There are 90 districts 
and we do IS every year. 

We have a requirement that all schools go through self evaluation - 
curriculum^ buildings and grounds^ administrative services eta. The 
evaluation unit prepares self-evaluation materials and visits the 
schools to explain them. A school has to prepare a five year plan and 
we check that they are following it. Thi plan goes to the accrediting 
aormCttee. All elementary schools and non-accredited high schools have 
to go through the self-evaluation process. There are two problems with 
this. Firstly^ it takes a lot of time for the local people to go 
through the self -evaluation process. They are given a year and they 
usually meet cmce a month. Secondly j community involvement is required 
and this is difficult to get. We have just gone through a 5 year cycle 
of self-evaluation. We pilot tested it for two years and got input 
frm the local people. Now we are undergoing a ^revision of the process. 
Principals and superintendents say that the process is very useful. We 
are looking at ways of cutting down time for self-evaluation. School 
boards feel that it is very useful. They often serve on the self-study 
committee. We would like assistance with how better to aggregate data 



*A11 quotes taken over the telephone are partly paraphrased. 
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from the reports in snah a way that we can better see what the problems 
of schools are and the state can then respond to these problems. 
From several hundred reports it is difficult to discern what the problems 
are. Self^study leads to action on the part of those doing it. 

vFour states indicated that they used no particular strategy but 

employed the strategy that was most suited to the program being 

evaluated. 

We use no particular strategy. We tailor the design to the program 
needs. We do not advocate one methodology over another. We are responsive 
to our target oMiienaes. 

other evaluation strategies mentioned were as follows: 

(2)'* - Case studies. 

(2) - Strategies oriented towards the needs of decision 
makers * 

(2) - Monitoring programs. 

Our evaluations basically involve monitoring programs to see if the 
programs are doing what they said they were going to do. 

(2) - Monitoring evaluations* 

LEAs prepare their evaluations for us. We give them guidelines 
which are based on RMC Model A of Title I. I also conduct workshops 
giving technical assistance for these evaluations. 

(1) - Involve program personnel in all stages of an 
evaluation. 

(1) - Adversary approach* 

(!) - Encourage the use of Title I designs. 
(1) - A three-member monitoring team. 

A three-member monitoring team goes to a site visit for two days. 
The leader is a college professor whh writes the report^ The second 
member is an expert in the substantive area the project to be evaluated 
is concermed with. The third person of the team is a mender of the 
advisory council. This team reports back to the advisory council which 
has oversight of all projects. The resultant evaluation report is valued 
by the council because it covers aspects of a project that cannot be 



♦Numbers in parentheses refer to the number of states involved. 
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measured. The x^eport aomptements the evaluation report based <m a 
quasi-experimental design. 

In relation to the question, "Is there a difference in the methods 

used in the evaluations your unit does versus those studies you 

contract out?" ten states could not respond since they did not contract 

out evaluations. Three states said there was no difference. There 

were varied replies from other states, some of which are as follows: 

Our evaluations are different from Title I evaluations (the 
contracted evaluations) which are Just information collecting. 

The contractor does not have the same concern about the program 
that we do'^that it is going to work. That is why we engage in program 
management as well as program evaluation. 

Contract evaluations are sumnative with a pretest-posttest design 
cmd we attempt to backfill these evaludtions with formative evaluation, * 

The contracted evaluator is usually a substantive expert. Usually 
he does interviewing and obserroing and thus his evaluation is usually 
more informal and less structured than ours* 

Yes, but it is really due to the nature of the contracted studies, 
not due to any difference in methodological philosophy. Contracted 
studies tend to be suimative. 

In summary, some states indicated using more than one strategy. 

By far the majority of states (eleven) used a behavioral objectives 

approach. Associated with this approach was the use of tasting and the 

use of an experimental or quasi-experimental design. Pour states used 

an auditing-accrediting approach to evaluation. Pour states indicated 

that they used no particular strategy but employed the strategy that 

was most suited to the program being evaluated. In addition a variety 

of other strategies were mentioned. Por those states that contracted 

out evaluations, most said thare was some difference between contracted 

evaluations and evaluations which they did. 



*By formative evaluation, the unit head appeared to mean process 
evaluation. 
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The problems and constxaints of carrying out evaluations , in 
relation to the question, "What problems and constraints do you 
experience in carrying out evaluations?" most states mentioned more 
than one. There was a great variety of answers. 

(7) - Shortage of evaiuators and/or too many evaluations. 

(6) - Costs mentioned were for personnel, travel, 
accommodation and food . 

(5) - Lack of time . 

(4) - Lack of training. Three states indicated that lack 
cf knowledge of evaluation was a deficiency. Other 
areas of training mentioned were lack of knowledge 
of experimental design and statistical analysis, 
and lack of knowledge of the substantive area 
being evaluated. 

(3) - The difficulty of setting up an experimental design. 

The following is a list of other problems which were described. 
They have been divided into problems related to carrying out evaluation, 
problems of evaluative impact. 

PROBLEMS RELATED TO CARRYING OUT AN EVALUATION 

Lay persons assume that it is possible to answer questions in the 
social soiences like it is in the physical sciences. They assume^ for 
example, that we can make causal connections where this is not possible. 
For example, a typical urban school will receive fimds from multiple 
sources for multiple reasoTis, A child is receiving his education from 
multiple funding sources. It is difficult to partial out the effects 
of any one set of funds— to attribute any change to one set of funds. 

^ Projects are planned independently of ow? involvement. l-/e like to 
be involved in the planning of a project. We would like inservice 
training of project people on planning and evaluation. 

The problem of defining the questions. After a set of questions 
have been developed, program persons will decide that they want other 
questions addressed. After an evaluation they will also decide that 
other questions should have been addressed. This effect is a result of 
the growth of their perceptiveness. 

Schools obtain multiple requests for data-^jrom federal and state 
agencies and from universities. It is difficult for a school to be 
responsive to all these requests. Thus at times it is difficult to get 
the data one requires from schools. 
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In the state -uide assessment survey we are seen as inundating 
districts with requests for information. Then we have to get tough 
though we prefer not to. The reason why districts are inundated is 
that in the districts close to the major urban area there are four 
institutions of teacher education. People want to collect information 
from the schools for Masters and Ph.D. research. In the smalt school 
districts, the administrative staff is smll and is overloaded with 
work and so does not respond easily to information requests. 

Evaluation is post hoc and it is difficult to establish a base 

line, 

Sound sampling frames are not available. 

Our problem is data collection. Data is only as good as the local 
schools send us. We need an improved data management system. 

We have a small staff. There are $30 LEAs in our state. The LEAs 
do the actual evaluation. Personnel in the LEAs lack training in 
evaluation. For example, they think testing and evaluation are 
synonymous. They fear evaluation because they see it as accountability. 
They lack knowledge of testing and evaluation strategies. 

PROBLEMS WITH REPORTING 

There is a problem of producing a report that is readable by a 
variety of audiences. Different audiences have different needs. 

We have trouble translating the evaluation results to program 
administrators so that action will be taken. 

Problems with incorrect data in the Title I reports done by LEAs. 
They have not read the directions and there are typographical errors. 

PROBLEMS WITH THE IMPACT OF EVALUATION 

We would like to see the results of evaluations used more 
extensively. We would like to see change as a result of evaluation. 

We don't see implementation or action taken on evaluation findings. 
We intend to do more followup to see if action has been taken. We often 
involve the program people in writing the recommendations and as a 
result action is more likely to be taken. 

It is difficult to know what information to collect that will be 
useful to decision makers. It is also difficult to know when they will 
want the information to make decisions. 

The problem of making school personnel realize the importance of 
evaluation and getting them involved in evaluation. 

^ Credibility. There is a lack of trust of the department in conduce 
tvng a survey and therefore we have to contract out. 
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Identifying evaluation studies which are aritical as opposed to 
routine, A lot of evaluation studies are not realty vnportcp2t. We don't 
get to do the evaluations we should be doing. This is because 
evaluations are threatening. People don't want the truths positive or 
negative about a program. 

It is difficult to get people to think rigorously about evaluation. 
They do not see it as essential but something that is a requirement 
that is laid on them by state and federal agencies. We would like them 
to get to see the utility of evaluation, 

states which used an auditing-accr editing strategy experienced 

problems peculiar to the strategy. One state indicated that visiting 

schools took a lot of staff time and that it was physically and 

emotionally wearing. There was also a problem with the standards set 

by the states. 

The minimum standards are general standards and are thus open to 
interpretation. There is a difference of interpretation within the 
evaluation staff and also a difference between the district personnel 
and the evaluators. 

Another state indicated that evaluators had not previously visited 
the schools and were finding it difficult to adopt the auditing role. 
The state which emphasized self-evaluation in the auditing of schools, 
indicated that firstly the schools found it time consuming and secondly, 
community involvement, which is required, is often difficult to get. 
The state was also finding it difficult to aggregate data from across 
schools in order to see what common problems they experienced. 

In summary, units gave a wide range of answers to the question of 
vdiat probleHu. and constraints they experience in carrying out evaluation. 
The commonest problems were shortage of evaluators and/or too many 
evaluations, costs and lack of time. 

Strategies used in planning an evaluation . In relation to the 
question, "Are there any specific techniques or strategies that you use 
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in planning an evaluation? " the following answers were obtained: 
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(9) - Attempt to be responsive to the audiences' needs. 

try to plan so that information hsill he oolleoted that is 
useful to the program organisers, the superintendent and the funding 
source » 

^In planning we first of all generate questions to be answered some 

of these questions are mandated and others we generate ourselves. 
There is a series of meetings with program managers and we determine 
what questions they haoe, especially formative ones. 

First we review the written document that relates to the program 
and review the objectives. We then meet with the program manager and 
verify that these are the objectives. We then ask the program manager 
if he has additional questions that he would like addressed in the 
evali4ation. We then ask the unit director ^ the assistant commissioner 
and the commissioner vliat questions they would like addressed. We get 
more data than is required by the funding source. We ask for 
evaluation questions from anybody in any way connected with the program. 

We seek to involve the client in the evaluation such as the sc^ 70I 
district people. From department administrators we also obtain questions 
cmd issues. We also may add questions. We try to be responHve. This 
is important if people are going to use the data. We interotit with the 
local district people to see haw our evaluation can interface with 
their decision making needs. 

The states indicated that their planning strategy was used in 

order to make the evaluation useful and to have impact. 

It is important that program perso^vtel obtain a sense of ownership 
of the evaluation and hence they are likely to take action on it, 

(4) - Use the planning strategy associated with their 
auditing-accrediting approach. 

(4) - Identify objectives and then design the evaluation. 
Two states indicated that helping program personnel 
to state their goals was useful to prograjn 
personnel in planning their program and it also 
made the program more capable of being evaluated^ 

We attempt to get the evaluation planned at the same time that the 
program is planned. We try to find out what the intentions of the program 
p tanners are and help them to state their goals. This helpa to make it 
possible to evaluate the program. 

(3) - Use no particular strategy since it depends on the 
evaluation . 

(1) - Use RMC Modal A. 
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In sunonary^ the conanonest strategy used in planning an evaluation 
was one that was aimed at making the evaluation responsive to the 
audiences' needs • Such an evaluation was seen as more likely to be 
useful and to have impact. 

Strategies in collecting data . In answer to the question ''Are 
there any specific techniques or strategies that you use in collecting 
data? " Mauiy states mentioned more than one technique* 

(12) - Testing. 

(3) ^ Interviews. 

{ 8) - Classroom observation. 

( 5) - Whatever data collecting strategy is appropriate to the 
particular evalxiation. 

{ 4) - Use some kind of document or record. For eacample one 
state uses the application form that schools fill out 
for what they are going to do for Title I as a source 
of data. Another state uses the individual education 
program for each child. 

( 3) - Questioxmaires. 

( 1) - Uses a large, once-a-year collection of data from all 
schools. This has the effect of cutting down the 
amount of d:stxirbance to each school. 

In reply to the question^ "VJhat factors determine your choice of 

(data collecting) technique or strategy most states indicated that it 

depended on the particular program being evaluated. Utility, credibility, 

and time were mentioned as factors. Other answers were: 

We use the strategy which will give the hardest, most objective 
data, otherwise an evaluation degenerates down to the level of opinion. 

Interviewing and olasoroom observation is better than hard 
statistical data. 

Not wanting to disturb the school with our data collecting. 

In siammary, a wide range of data collecting techi.. jues were used, 

the commonest being testing, interviews and classroom observation. 

ERLC ' 



Methods used in reporting an evaluation . All states produced 



written reports of evaluations. Reasons given for using written reports 
were: 

Writt&n reports are required by clients. 

Don't have a ahanae to talk to •program personnel. We produce two 
written reports "'■one for the lay person^ such as program personnel, and 
a technical report* 

Oral reporting is limiting — it only becomes somebody's recollection. 
That is why we always use written reports. 

All evaluations J, even those that are federally funded have to be 
reported back to the legislators. Witten reports are necessary. The 
state department has to forward the reports to the state board which 
forwards them to the legislators. 

Written reports are required to meet OE requirements. 

In addition to the use of formal written reports, five states also 

mentioned the use of written press releases. Because of public interest, 

one state said they had press conferences where there was a press 

release, an oral presentation and the use of pictorial material. The 

oral presentation was often done by the superintendent. At least eleven 

states inentioned giving some kind of oral repoirt. 

We have a reporting session with the people involved in the program. 
We discuss with them the questions of tke evaluations how to interpret 
the statistics and we assist them in making inferences. This procedure 
aids in the utilization of the evaluation results. 

We give an informal five^ to ten-^nute oral presentation to the 
cormCssioner and assistant comnissioners. We also give a more formal 
oral presentation to the state board of education. The aim of the oral 
presentation is to get commitment so action will be taken. 

An oral report will often have more effect than a BOO page written 
report. The audience is more likely to take action. 

Most states appeared to maJce an effort to report in a way that best 

suited their audiences in order that impact would be maximized. Thus, 

states often used more than one type of written report and used a 

coiobination of written and oral reports. 
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We try to identify audienaes and target reports for the audiences. 
On state assessment for example^ we use specific reports for specific 
audiences^ 

We prepare three reports - a one page simnary^ a limited page 
executive sumnary (three to five page^), and a total technical report. 
We always produce written reports. We also give informal and formal 
oral reports to the appropriate client. 

Firsts we do an exit interview where the evaluation staff meet 
with the program administrators. Findings are present and interpreted. 
We link the result giving with the next evaluation cycle. We get 
questions and issues for the next cycle. Secondly^ there is a written 
report which presents data and interpretations. Thirdly^ there is an 
executive swmary written in terms the layman can understand. This is 
given to the state board of education. We also at times wilt hold a 
press conference eg. for state assessment. 

In summary, all states produced written reports of evaluations. 
In addition eleven states mentioned giving some kind of oral report • 

Are there evaluations that you have to do for which you have 
inadequate techniques or strategies? Thirteen states answered in the 
negative. The positive replies were quite varied as the following 
illustrate . 

At times we have trouble producing suitable tests. 

We are unable to aggregate information so one can determine gains 
sidch as in Title I evaluations. Another problem is that if information 
from 300 schools (say) is aggregated^ there is often no significant 
differences. However^ the no significant differences ignores the fact 
that there are both good and bad programs. Consequently^ we use case 
studies in order to daaument the programs. Another problem is that what 
is lacking is a set of criteria or categories for judging a program. 
When Consumer Reports investigates a product such as a new car^ they have 
a checklist of criteria for judging the product. 

Virtually all evaluations. In the beginning we did evaluations 
based on the social science research model which was a scientific model. 
There were enticing words like control group. The model attempted to 
get at causes. We pretended for a long time that this was the answer 
and lay persons came to expect this approach to evaluation. Now we are 
trying some process and case study evaluations using a modified form of 
Scriven^s modus operandi approach. 

We are not satisfied with Title IV designs. 
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Axe there certain information needs of decision makers for which 



you feel that you have inadequate strategies or techniques? 
(10) - Answered in the negative. 

{ 4) - The difficulty of obtaining information, including 
a suitable management information system. 

We do' not haoe a oomprehensive oolteation of information (a data 
base) on which to base planning. 

We have massive data bases which ace stilt hand tabulated. The 
legislature asks questions and as a result we take a tang time to answer 
the questions. We need an improved data management system. 

( 2) - The problem of partialing out the effects of 
separate sources of funds when there are 
multiple sources of funds. 

In showing program irnpaat, for example in the eoaluation of Title I 
by the models developed by OE and BMC. It is difficult to try to partial 
.jeMt.the eff-aats of Xitte. I ,fimds. For example we haoe students where 
the reading instruction is funded by multiple sources — local, state, 
migrant ee£ication, eta. It is difficult to partial the effects of these 
multiple sources of funds, yet policy makers want to know the impact of 
a single source of funds. 

Other answers to the question were vauried. 

It is difficult to get a clear delineation of what decision makers 
want in the way of information. People cannot predict their needs for 
information welt. 

There is no formal way of approaching the legislature. The maKe 
deoisiona without information. They don't ask for evaluation. 

Are there evaluation techniques or strategies that you are required 

to use that you think are unsuitable? 

(14) - Answered in the negative. 

( 5) - The inadequacy of Title I evaluations. 

Title I evaluations using NCEs (Normal Curve Equivalents) . They 
^e not sensitive enough for the measurement of change. 

Concerned with Title I reporting model'^-there is over inflation of 
results. We see Model C as most technically sound. Model A is easier 
but it aver inflates the results, 

{ 1) - Have to use statistical significance but this is 

difficult to translate into educational significance. 
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(1) Difficult to set up experimental and quasi-experimental 
designs, especially when random assignment is required 



Are there any new techniques or strategies that you would like to 



(7) - Answered in the negative. 

(4) - Interested in trying new techniques but could not 
be specific. 

(2) - Goal-free eval\:ation. One of these states said that 
it would be time-consuming but would produce higher 
quality data for decision making. 

(2) - Self -evaluation techniques. 



(1) - Techniques to assess affective outcomes. 

(1) - "Big computer techniques" such as multivariate 
analysis . 

Other answers to the questions were as follows; 

Some of our evalmtora are interested in the descriptive, anthro- 
pologiaal approach and the use of case studies. We are interested in 
innovative measurement procedures especially for those things it is 
important to measure. 

The use of unobtrusive measures to get at such things as school 
climate. The climate in a school is very important to the achievement 
of the student. It is very difficult and time consuming to get at. 

We would like to try techniques and strategies that are more 
research oriented in the sense that we answer "why" questions about a 
program^going beyond a simple Judgment of worth. For example, we are 
using the modus operandi approach. 

What Kind of Assistance (If Any) With Techniques or Strategie s Would 
You Like? ~* 

Twelve states indicated that they were not in need of amy 
assistance. The following is a list of areas that other states would 
like assistance on. There was no area that was indicated in common by 
tvK> or more states. 

1. Assessing affective outcomes. 

2. Checklists of criteria for judging the worth of a program. 
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3. How to report in a way that is brief out gets the 
important points across. 

4. How to make site visits inore productive. 

5. Streamlined ways of getting information ready for 
punching and processing. 

6. Assistance with goal-free evaluation. 

7. We would like inservice training on new techniques. 

8. Advice on the suitability and validity of test instr\Ments. 

9. In the area of sampling. How to senile student and public 
opinions . 

10. Techniques for presenting data in an interesting way. 

11. How to present data to lay persons. 

12. Any kind of instruments that people have used for self 
study. 

Summary of Results 

Of the 25 state education departments contacted, four indicated 
that they did not have units that actually conducted program evaluation. 
Thus data was collected on only 21 state evaluation units. 

It is clear from the data that the SEA evaluation units are 
extremely diverse both in their nature and the activities they carry out. 
I'hey vary widely in the percentage of time they 5u:e involved in program 
evaluation, the average number of evaluations conducted and contracted 
per year, and the size of their staffs. Some of the units are involved 
in a multitude of activities besides program evaluation. 

The most popular methodology (11 states) involved in identifying 
objectives and determining the extent to which they had been achieved. 
Associated with this approach was the use of testing. While they find 
it difficult, if at all possible, they attempt to set up an experimental 
or quasi-experimental design such as involving pre- and post- tasting and 
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a control group. Units help project managers and staff identify 
objectives and often assist in planning. By doing this, units aim to 
mak« projects more amenable to evaluation. Hence many units like to be 
invoived in the planning stage of an evaluation. Four states used an 
auditing-accr editing approach to evaluation. Pour states indicated that 
they used no particular strategy but ert^loyed the strategy that was most 
suited to the program be ing^ evaluated • Other than the coimnonalities 
mentioned above, there appears to be quite a diversity in the evaluative 
methodologies being used. 

Units gave a wide range of answers to the question of what 
problems and constraints they experience in carrying out an evaluation. 
The commonest problems were shortage of e valuators and/or too many 
evaluations, costs and lack of tiina» 

One theme emerging from the data is that relating to the impact of 
an evaluation. One of the problems mentioned by units was the lack of 
impact. One way units tried to increase impact was in the planning 
stage of an evaluation ♦ They try to plan so that the information will 
be useful to their various clients. They seek to involve the clients 
in the evaluative process. Evaluation questions are solicited from 
liXely audiences to the evaluation. Another way units seek to increase 
impact is in the reporting of evaluations. Most states appeared to make 
an effort to report in a way that best suited their audiences. Thus 
units often produced more than one type of written report, including a 
short report written in layman's terms. They also made use of oral 
reports to increase their impact. 

A wide range of data collecting techniques was used, the 
commonest being tasting, interviews and classroom observation • 
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Thirteen states answered negatively to the question of whether 
there are evaluations that they have to do for which they have 
inadequate techniques or strategies. The positive answers to this 
question were quite varied. 

Tan states answered negatively to the question of whether there 
are certain information needs of decision makers for which they have 
inadequate strategies or techniques. Four states said that they had 
an inadequate infornation system for answering the questions of decision 
makers. Two states said that there is a problem of partialing out the 
effects of separate sources of funds when there are multiple sources of 
funds. 

Fourteen states answered negatively to the question of whether 
there are evaluation techniques or strategies that they are required 
to use that they think are unsuitable. Five states mentioned the un- 
suitability of Title i designs. 

En answer to the question of whether there are any new techniques 
or strategies that they would like to try, seven states answered 
negatively. Pour states were interested in trying new techniques but 
were not specific. Two wanted to try goal- free evaluation and two 
wanted to try self --evaluation techniques. 

Twelve states felt that ihay did ncJt need assistance with techniques 
or strategies, other states mentioned a wide variety of areas in which 
they would like assistance. 
Implications of the Results 

One of the major purposes of the survey was to try to infer what 
might be the nature of innovative methodologies that would have utility 
to SEA evaluation units. The following inferences are mainly based on 

37 " 

ERIC 



the problems and constraints that SEA evaluation units experience in 
caurrying out evaluations • The problems and constraints experienced by 
a \init will vary with the dominant methodology employed. For example 
states that employ an auditing'-accr editing methodology will have 
problems that differ from states that mainly monitor evaluations. The 
following is a list of suggestions as to the nature of innovative 
methodologies that might have utility to SEA evaluation units. 

1. Shortage of evaluators and/or too many evaluations, and 

time are interrelated problems and were mentioned by a niamber of states. 
The time it takes to do otat evaluation is also related to costs • 
Thus a useful methodology would be one which enabled an evaluation to 
be carried out quickly and efficiently. Assuming time is money, this 
would reduce the cost of an evaluation and also enable more evaluations 
to be carried out. While we do not have a reference, we know that some 
British evaluators havr?. been experimenting with ways of maximizing the 
amount of evaluative information that can be gleaned by a single day's 
visit to a school. Of course the speed with which am evaluation can be 
done is paxtly dependent on the attributes of the evaluator. For example 
an evaluator is likely to be more efficient if he is familiar with the 
"culture" of schools and is an expert in the substantive area being 
evaluated. Cuba (1978b, p. 110) has reported on the use of the "fast 
study" in investigative journalism. 

2. Methodologically, many states attempt to set up a Can^bell 
and Stanley experimental design for an evaluation. There is a belief 
that this is the most rigorous methodology* However, they experience 
difficulties with setting up an experimental design. It is often 
difficult to set up control groups and to carry out random assignanents. 
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The number of students involved may be too small to make statistical 
inference, it is impossible to set up an experimental design when 
evaluators are called in after a program has been operating for some 
time. Clearly what is wanted is a m-Sithodology which is equally rigorous 
as the experimental design but does not have its drawbacks. Guba 
<1978a, p. 25) argues for the rigor of naturalistic inquiry "To provide 
an alternative where it is in^ssible to meet the technical assumptions 
of the experimental approach in the real world." However his study of 
the rigor of naturalistic inquiry is still in progress. 

3. Schools obtain multiple requests for data— from federal and 
state agencies and from universities. It is difficult for schools 

to be responsive to all these requests. It is clear that some states 
want a data management system. The nature of such a system would be 
that it avoids multiple requests for the same data, that it consumes 
as little time as possible of school personnel, that it produces as 
little disttarbance as possible to school operations and that it be 
centrally located. Such a system would mean that basic data about 
pupils and schools would be readily available to evaluators. Evaluation 
studies that required aanpling could be more easily carried out, 

4. Lack of impact of evaluations is one of the problems that 
clearly troubles states. Any methodology that increased in^jact would 
certainly be welcomed. Besides their reporting strategy states attempt 
to increase ia^ct through their planning strategy. This strategy is 
to obtain evaluative questions from a wide variety of audiences, 
particularly decision maJcers. The assumption behind this is that 
audiences will ask for information that will be useful to them and, if 
necessary, take action on it. However, as one unit head remarked: 
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"People cannot predict their needs for information well." Are 
there methodologies of planning and organizing an evaluation that would 
increase the impact of an evaluation? This is by no ineans a new 
question. 

5. It is clear that states make strenuous efforts so that their 
evaluation reports will have impact that will lead to action. Ways of 
reporting data so that they will have most impact is clearly desired. 
Perhaps techniques from journalism, advertising or graphic design could 
be adopted for this purpose. 

6. Poxir states use an audX ting-accrediting approach to 
evaluating schools. Methods to streamline this approach would be 
appropriate. This approach requires a lot of staff, is time-consuming 
and is physically and emotionally wearing. One state, which used self- 
evaluation by the schools as well as site visits, experienced the problem 
that schools found self -evaluations very time consuming. 

7. Some stv;tes do a considerable amount of monitorinq of 
evaluation which is carried out by the local school districts. The 
problem experienced with this approach is that school district personnel 
lack training in evaluation and are unable to follow the guidelines 
provided by the state evaluation unit. Solutions to this problem could 
be one or more of Uae following. 

{a) Train local district personnel in evaluation. 

(b) Improve the guidelines provided, 

(c) Produce simple models of evaluation that could be 
easily followed. 

(d) increase the support provided by the state unit. 

8. Since the problems and constraints of the state evaluation 
lanits are so diverse, a certain problem often applying only to one 
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state ^ a strategy to take would be to develop innovative methodologies 
and then determine what problems they solve. This is an alternative 
to taking the problem and trying to develop a methodology to solve that 
problem. 

In conclusion, one cannot say that the results of this survey of 
the methodology used by SEA evaluation units is at all surprising. One 
could probably do a survey of evaluators in general and come up with 
similar results. The behavioral objectives approach is the dominant 
methodology. Lack of time and tight budgets are typical enemies of 
the evaluator. The constant complaint of evaluators is about the lack 
of impact of their evaluations. But what special circumstances do SEA 
evaluators operate under? They are internal to the system they are 
evaluating, for unlike the external evaliaator they do not negotiate a 
contract for each evaluation they do. They are captive evaluators in 
the sense that they do not often have a choice whether or not to do an 
evaluation. Within a state education department, evaluation is often 
closely tied to planning. This telephone sxirvey is to some degree 
superficial. The next step would be to carry out intensive on-»site 
visits to examine SEA evaluation units. Hopefully this would produce 
deeper insights into their operations and reveal the peculiar problems 
they experience* 
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APPENDICES 



APPENDIX I 



Some Problems of the Office of Educational 
Evaluation of the City of New York 

Some of the problems mentioned by Polemeni (undated) , Director of 

the Office of Educational Evaluation, Board of Education of the City 

of New York, are a$ follows: 

. . . the major problems today concern responses to 
the evaluation report by school superiiit*»;id6...ts , 
principals, teachers, unions and parent groups. The 
evalixation report has become politicized and has become 
the source of problems which must be faced by 
evaluators as they attempt to protect their findings 
from special interest groups. For oximple, there is 
often an unwillingness to publicize results vrfiich, 
though truly illustrative of the situation, might 
have negative political repercussions. It might be 
found, for instance, that ability grouping produces 
maximum growth in academic achievement. If, however, 
ability grouping would result in racially segregated 
classrooms, than such findings would, in many 
localities, be considered political anathema. . . 

For a variety of reasons, evaluation results are 
often not considered in making program or 
management decisions: The results may be considered 
politically inexpedient; the results are available 
too late for incorporation in the recycling design; 
the person in a key management position singly does 
not agree, philosophically, with the resialts. Whatever 
the cause, the net result is the same; The evaluation 
might just as well never have been conducted. This 
situation gives rise to a commitment problem in that 
the morale of evaluation workers suffers when they 
become aware that their work may be in vain. As in 
most such situations, reduced morale results in 
reduced quality of output, . . . 

Laymen — particularly on Boards of Education — demand 
gross over-simplification in reports of evaluation 
studies. . , . the evaluator trying to stress the 
limitations of his study, the lay user trying to make 
the findings universally applicable to satisfy his 
own purpose. , . , 

There exist no effective channels of conmiunication 
between evaluators and field personnel and, as a 
consequence, research findings are seldom implemented. , . , 
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Frequently there is resistance to evaluation findings 
by supervisory personnel — including school principals— 
who "know in their bones" that the way they are doing 
it is the best way it can be done. This situation 
leads them to say such things as, "I know your findings 
prove my reading program is not working, but I feel 
the children are getting something out of it and I am 
going to stick with it*" • . . • 

Classroom teachers frequently oppose the collection of 
data because they are unable to see any profit to 
their own students. This is almost always the case 
with control groups from whom vast amounts of data 
must be taken without any program to compensate them 
for their time* In large degree their reluctance is 
well-founded since the period between data collection 
and report dissemination usually mns a year or more 
and the students who contribute the data are no 
longer with the teacher. . . . 

Project managers are threatened by evaluation since, 
if the evaluation is negative, they might be out of 
a job. Such consideration causes all sorts of things 
to happen: data disappear, project personnel are 
unavailable for interview, students to be observed 
have suddenly gone on a class trip, the evaluator 
is incompetent, the evaluator is biased, and so forth, 
and so forth. While it is not absolutely impossible 
to evaluate a program without the project manager's 
approval, it is extremely difficvilt. . . . 

School people are^ increasingly, demanding that if 
evaluator s are going to collect the data, they also 
provide a prescription for upgrading the program. . . . 

Still another vice for the unwary is the one that 
says "Good news won't sell papers." This pithy 
aphorism probably accounts for the media's propensity 
to simplify or alter research and evaluation data. 
No matter how much the scores might have gone up 
within the city, there's always one school where they 
went down. Let's run that headline, Charlie. • . . 
What is needed is a balanced presentation; nothing 
is either ail good or all bad. . . . 

There is frequently a tremendous disparity between 
the amount of evaluation data required and the 
amoimt of funding provided for tlie conduct of the 
evaluation. Thin happens most often where a figure 
such as one-half of one percent is determined to be 
appropriate for the evaluation of each individual 
program. Where the program costs run to several 
hundred thousand dollars or more, this base rate is 
totally meaningful and applicable. Where the cost 
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of a program runs to figures like thirty thousand 
dolisurs, the evaluation agency is being requested to 
perform its function for about one hundred andi . fifty 
dollars — which is absurd on the face of it. 
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APPENDIX II 



Structured Interview Schedule for a Telephone Survey of SEA 
Administrators of Evaluation Units 

Name Phone No. 

Neune of State 

Presumably through secretary, identify and contact above person. 

Sayi* 

HELLO. THIS IS DARPBL CAULLEY OF THE NORTHWEST REGIONAL 
EDUCATIONAL LABORATORY IN PORTLAND, OREGON. *WE HAVE A 
FEDERAL GRANT TO DEVELOP NEW EVALUATION METHODS FOR LEA AND 
SEA EVALUATIONS. AS A PART OF THAT WORK WE ARE INTERVIEWING 
A FEW SENIOR ADMINISTRATORS IN LEAS AND SEAS TO BETTEJl 
UNDERSTAND THE PROBLEMS AND CONSTRAINTS THEIR STAFFS 
EXPERIENCE IN CARRYING OUT EVALUATIONS. I CAN ALSO TELL 
YOU A LITTLE ABOUT THE NEW KINDS OF METHODS WE ARE DEVELOPING. 

SINCE WE ARE TRYING TO DEVELOP NEW METHODS THAT CAN BE OF 
USE IN OPERATIONS LIKE YOURS, I'D LIKE TO ASK YOU A FEW 
QUESTIONS ABOUT IT. THESE QUESTIOIS SHOULD TAKE NO LONGER 
THAN 20 MINUTES. DO YOU HAVE THE TIME NOW OR SHOULD I CALL 
BACK LATER TODAY? (if late in the day, say EARLY TOMORROW 
MORNING) 

CALL BACK: HI, THIS IS DARREL CAULLEY OF THE NORTHWEST REGIONAL 

EDUCATIONAL LAB. I TALKED WITH YOU AND YOU ASKED ME 

TO CALL BACK. AS I MENTIONED BEFORE, WE (to *) . . . . 

♦Only the words in upper case are to be spoken over the telephone. 



DATE 


TIME 


RESULT 









Examples of Results: 

!• Not in office • Will return at a certain time 

2^ Busy-Inconvenient. Phone back at specified time. 

3« Refused. 

4. Interview ccmipleted. 
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A.1 «RAT PERCENT OP THE TIME DOES YOUR STAFF SPEND IN 

DOING OR CQNTSACTING PROGRAM EVALUATION STUDIES LIKE 

TITLE I EVALUATIONS OR EVALUATIONS OF, FOR EXAMPLE 

READING PROGRAMS? ^% 

A.2 IN GENERAL, WHAT OTHER ACTIVITIES DOES YOUR UNIT 

ENGAGE IN? 

% 



% 



% 



% 



ON THE AVERAGE, APPROXIMATELY HOW MANY EVALUATIONS 
DO YOU CONDUCT IN A YEAR? 



HOW MANY EVALUATIONS DO YOU CONTRACT? 



LARGE IS YOUR STAFF? Full-time equivalents 



A. 3 (a) 

(b) 

A. 4 HOW 
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A* 5 (i) WHAT STRATEGIES (OR DESIGN) DC YOU USE IN DOING AN 
EVALUATION? 

Rank 

Possible Answers: Order 

(a) Identify objectives and determine the 

extent they have been achieved 



(b) Do an experimental or quasi-experimental 
design, e.g^r pretest-posttestr 
e3^erimental**control group 

(c) Testing students to see how they compare 
to norms 

(d) Do a sxirvey using a questionnaire 

(e) Do interviewing and/or observations to 
find out how a program is working 

(f) Other 



(g) Other 



(ii) COULD YOU PLEASE RANK THE STRATEGIES IN ORDER FROM T. Z 
MDST USED TO LEAST USED. 

On the above place the rank using 1 for the most used. 

(iii) IS THERE A DIFFERENCE IN THE METHODS USED IN THE EVALUATIONS 
YOUR UNIT DOES VERSUS THOSE STUDIES YOU CONTRACT OUT? 
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WHAT PROBLEMS AND CONSTRAINTS DO YOU EXPERIENCE IN CARRYING 
OUT EVALUATIONS? 

Possible Answers J 
Ci) Time 

WHAT ASPECTS OF EVALUATIONS ARE TIME CONSUMING? 

(a) Planning the evaluation 

(b) Collecting data 

(c) Analyzing data 

(d) Writing reports 

(e) Travel to and from site 

(f) Other 



(ii) Cost 

WHAT ARE YOUR MAJOR COSTS? 
(a) Personnel 
(bi Travel 

(c) Accommodation and food 

(d) Costs of analyzing data 

(e) Costs of printing reports 

(f) Acquisiton of tests 

(g) Other ^ 



(iii) shortage of evaluators and/or too many evaluations 
PLEASE ELABORATE 



0 



51 



(iv) Lack of training of personnel 

WHAT ASSAS OP TRAINING DO THEY LACK? 



Instructional product development 

Plamning 

Management 

Conjmunications 

Operations research 

Econometrics 

Path analysis 

Cost effectiveness 

Case study 

Historical research 

S}<periniental design 

Library research 

Computer programming . 

Canned program usage 

Other 



Stochastic processes 
Bayesian analysis 
Multivariate inferential stat 
Univariate inferential stat 
Multivariate descriptive stat 
Univariate descriptive stat 
Politics of evaluation 
Objectives development 
Evalxiation theory 
Evaluation methodology 
Instrument development 
Scaling 

Measurement theory 
Testing applications 

Other 



Cv) Lack of cooperation 
FROM WHOM? 

School superintendents 

Principals 

Teachers 

Students 

Parents 

Other 

VIHAT FORM DOES THE LACK OF COOPERATION TAKE AND 
WHAT ARE THE REASONS FOR IT? 



(vi) Other 



(vii) Other 



C. 1. 



ABE THERE ANY SPECIFIC TECHNIQUES OR STRATEGIES THAT YOU 
USE IN PLANNING AN EVALUATION? 



WHAT FACTORS DETERMINE YOUR CHOICE OF TECHNIQUE OR STRATEGY? 



Possible Answers: 



(a) Utility 

(b) Credibility 

(c) Time 

(d) Costs 

(e) Manpower availability 

(f) Skills of the evaluator 

{g) Ease of obtaining cooperation 

(h) Other 



2. ARE THERE ANY SPECIFIC TECHNIQUES OR STRATEGIES THAT 
YOU USE IN COLLECTING DATA? 

Possible Answers J 



(a) Testing 

(b) Classroom observation 

(c) Questionnaires 

(d) Interviews 

(e) Other 



(f) Other 



WHAT FACTORS DETERMINE YOUR CHOICE OF TECHNIQUE OR 
STRATEGY? 



Possible Answers: 



(a) Utility 

(b) Credibility 

(c) Time 

(d) costs 

(e) Manpower availability 

(f) Skills of the evaluator 

(g) Ease of obtaining cooperation 

(h) Base of analysis 

(i) Other 
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3. (i) WHAT METHODS DO VOU USE IN REPORTING AN EVALUATION? 



Possible Answers: 

(a) Written 

(b) Oral 

(c) Pictorial 

(d) Other 

(e) Combination o£ the above 



(ii) WHICH TWO METHODS DO YOU MOST USE? 



(iii) WHAT FACTORS DETERMINE YOUR CHOICE OP METHOD? 



ARE THERE EVALUATIONS THAT YOU HAVE TO DO FOR WHICH YOU FEEL YOU 
HAVE INADEQUATE TECHNIQUES OR STRATEGIES? GIVE EXAMPLES. 



ARE THERE CERTAIN INFORMATION NEEDS OF DECISION MAKERS FOR WHICH 
YOU PEEL THAT YOU HAVE INADEQUATE STRATEGIES OR TECHNIQUES? GIVE 
EXAMPLES. 
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ARE THERE EVALUATION TECHNIQUES OR STRATEGIES THAT YOU ARE REQUIRED 
TO USE THAT YOU THINK ARE UNSUITABLE? GIVE EXAMPLES. 



ARE THERE ANY NEW TECHNIQUES OR STRATEGIES THAT YOU WOULD LIKE TO 
TRY? 



WHAT KIND OP ASSISTANCE (IF ANY) WITH TECHNIQUES OR STRATEGIES 
WOULD YOU LIKE? 



WHAT SHOULD WE KNOW ABOUT YOUR EVALUATION TECHNIQUES OR STRATEGIES 
THAT WE DID NOT ASK? 



WE APPRECIATE YOUR COOPERATION IN ANSWERING THESE QUESTIONS. YOUR 
NAME OR UNIT WILL NOT BE IDENTIFIED IN ANY RTJ^ORT OF THE RESULTS, BUT 
I WOULD BE PLEASED TO SEND YOU A COPY OF THE RESULTS WHEN THE STUDY IS 
COMPLETED. ALSO, IP YOU WOULD LIKE, I COULD PLACE YOUR NAME ON OUR 
PERMANENT PROGRAM MAILING LIST TO RECEIVE OUR NEWSLETTER AND OTHER 
NOTICES OF NEW METHODS. 

WOULD YOU LIKE A COPY OF THE SURVEY RESULTS? 
No Yes* 

WOULD YOU LIKE TO BE ON OUR MAILING LIST? 
No Yes* 

*Address 



THANK YOU AGAIN. GOOD BYE. 
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